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Abstract. We present a method for recovering the structure of a plant 
directly from a small set of widely-spaced images. Structure recovery is 
more complex than shape estimation, but the resulting structure esti¬ 
mate is more closely related to phenotype than is a 3D geometric model. 

The method we propose is applicable to a wide variety of plants, but is 
demonstrated on wheat. Wheat is made up of thin elements with few 
identifiable features, making it difficult to analyse using standard fea¬ 
ture matching techniques. Our method instead analyses the structure 
of plants using only their silhouettes. We employ a generate-and-test 
method, using a database of manually modelled leaves and a model for 
their composition to synthesise plausible plant structures which are eval¬ 
uated against the images. The method is capable of efficiently recovering 
accurate estimates of plant structure in a wide variety of imaging sce¬ 
narios, with no manual intervention. 

Keywords: Plant phenotyping, image processing, Plant architecture 

1 Introduction 

Computer vision techniques can provide fast, accurate, automated, and nonin- 
vasive measurements of phenotypic properties of plants. Measurements of prop¬ 
erties such as volume, leaf length, and leaf angle can be used to evaluate the 
effect on plants of variation in environmental conditions or genetic properties 
[U. Obtaining these measurements from image data can be difficult. Plants typi¬ 
cally have properties such as uniform colour, specular surfaces, and thin regions 
which present challenges for typical reconstruction techniques. 

Existing methods have focused largely on the problem of recovering the shape 
of the plant independent its structure. Structure, here, is intended to encompass 


2 


B. Ward et al. 


the various parts of a plant and the relationships between them, as opposed to 
purely geometric shape information contained in a representation of the whole 
plant such as a 3D volume or point cloud. The structure may be represented in 
terms of anatomical aspects of the plant, but may equally be described in terms 
of more basic elements. Importantly, structure allows the application of prior 
knowledge about the grammar of particular types of plants. 

There are two primary advantages of considering structure rather than shape. 
The first is that structure is much more closely related to plant anatomy, and 
therefore a much better indication of phenotype. The second advantage is that 
the structural properties of a plant provide a strong indication of the likelihood 
of a particular shape, which is a valuable cue when trying to select from among 
multiple feasible shapes. For plants with potentially complex structures, such as 
wheat, there may be many possible plant shape hypotheses which are supported 
by an image set, whereas prior knowledge of plant anatomy may indicate that 
only one structure is feasible. This means that structure recovery is possible 
when shape estimation alone would be ambiguous, or equivalently, that fewer 
cameras are required to estimate structure than shape. A related advantage is 
that even if more than one shape is supported by the image set, these shapes 
often have closely related structures, so although the images are ambiguous (in 
terms of shape) they may still support an estimate of structure, and thus a 
phenotypic interpretation. 

Structure, for the purposes of the method we propose here, includes infor¬ 
mation about the identity, length, and curvature of each leaf in the plant, and 
the relationships between leaves. In this method each leaf is represented by a 3D 
curve tracing the central axis of the leaf from its tip to the base of the plant. 
The combination of multiple leaf models gives a model of a complete plant. This 
estimate of structure implies a particular 3D shape of the plant, which may be 
used to estimate which pixels belong to each plant element. Estimating structure 
thus enables the length of leaf 4 on day 10 to be measured, for example, and 
post-processing would allow an estimate of the width or the length of senescence. 

The method we describe is capable of estimating the structure of a plant 
made up of thin elements from a small set of images taken from widely-spaced 
viewpoints. Because the properties of these plants make reconstruction difficult 
using standard feature matching techniques, we reconstruct the plants using 
only their silhouettes. We employ a generate-and-test method, generating pos¬ 
sible plant structures which are evaluated against the images. The generation 
process makes use of a database of leaf models, providing prior information on 
plausible leaf curves, which we use to restrict the generated models to plausible 
plant structures. The space of possible generated models is therefore significantly 
smaller than if we were to generate models by naively sampling 3D curves, allow¬ 
ing for a more efficient reconstruction process. Likely leaf tip locations are also 
detected, and used to further constrain the space of possible models. Figure [l] 
shows a 3D plant model estimated with this method projected into the original 
image set. 
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Fig. 1 . A reconstructed plant model projected into the original images, and another 
view of the 3D model 


2 Related Work 

A range of techniques currently exist for automated extraction of phenotypic 
properties from image or depth data. Highly detailed and accurate point-cloud 
reconstructions can be obtained with the use of technology such as laser scan¬ 
ners [2| or structured light [3I4| . However, this technology can be prohibitively 
expensive or infeasible to incorporate into existing systems, may not provide 
sufficient resolution for recovering thin structures, and can be difficult to apply 
when plant size varies greatly. Reconstruction from images can provide a lower 
cost and a more practical solution. Methods based on identifying plant pixels 
can be used to estimate volume without recovering 3D structure |5l6| . Image 
based approaches for recovering 3D reconstructions employ techniques such as 
dynamic programming [7] and simulated annealing [8] to overcome the difficulty 
of identifying corresponding points between frames. Reconstruction based on 
matching line features can provide robustness to appearance variation in differ¬ 
ent views mm- Complex plant structures with overlapping leaves mean a 
large number of views of the plant are usually required for a complete recon¬ 
struction. Techniques for obtaining a dense set of views of a plant include the 
use of mirrors [12] or cameras mounted on robotic arms m ■ Mechanisms for 
turning the plants [5] can be used to generate a range of views, but can cause leaf 
movement which leads to additional difficulties for reconstruction. For methods 
which recover a point cloud or volumetric description of a plant, additional pro¬ 
cessing such as applying skeletonisation operations to a point cloud is required 
to recover a structural description M- Interactive methods avoid some of the 
difficulty of fully automated techniques mm but significantly increase the time 
and manual effort required for reconstruction. 

3 Method 

We aim to recover an estimate of the length, curvature, and identity of each leaf 
of a grass plant, in this case wheat, from a set of images. The image set may be 
small (the results in this paper were obtained from four images), and captured 
with widely-spaced cameras. Widely-spaced views, and the thin components and 
relatively uniform colour of these plants, make accurate reconstruction infeasible 
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Fig. 2. A visual hull reconstruction illustrating the spurious shapes beyond the true 
plant reconstruction which are inherent to the visual hull. 


using standard feature matching techniques. Such techniques would also not 
provide data on the structure of the plant in occluded regions. 

Given that attempting to match the appearance of individual points on leaves 
is infeasible, we instead analyse the silhouette of the plant in each view. Using 
standard silhouette-based reconstruction methods El could leave the 3D struc¬ 
ture ambiguous when only a small number of views is available. Figure [2] shows 
three views of the visual hull generated from the four silhouettes for the plant 
on the left. Due to self occlusion and the limited set of views, this visual hull 
reconstruction includes leaf-like regions which do not correspond to actual leaves 
of the plant. 

Instead of directly recovering the 3D shape of the plant from the silhouettes, 
we use a generate-and-test method to recover the 3D structure, generating plau¬ 
sible 3D plant models and evaluating them against the image set. A process 
of using prior knowledge to generate plausible structures which are evaluated 
against the data is employed for tree and plant reconstruction by methods such 
as [18119120] . 

This method allows us to use prior information about the plants being re¬ 
constructed to aid in determining the structure in regions where that structure 
would be ambiguous given only the image data. We make use of a database of 
manually modelled leaves. The reconstruction process generates 3D plant models 
by finding leaf models in the database which closely match the current image set, 
then refining these individual leaf models, and selecting an optimal combination 
of leaf models to model the complete plant. 


3.1 Input Data 

The input to our process is a set of images of a plant. The method is suitable 
for use with any number and placement of cameras, provided two views are 
available for each leaf. For results in this paper, we used four images captured 
by cameras covering 360° around the plant. These images were captured with a 
set of consumer-grade DSLR cameras. The method requires calibrated cameras 
with known scale. We also require the approximate location of the centre of 
the pot, and a vector giving the vertical orientation of the pot. To obtain the 
necessary calibration information with minimal manual intervention, we make 
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use of a calibration object providing features on multiple planes in each view. 
A 3D model giving the approximate structure of the pot and pot holder is also 
used to estimate occlusion. We require that the leaves are static while images are 
being captured, and that the leaves do not move between images being captured. 

The structure recovery process estimates a silhouette of the plant for each 
frame. Depending on the background of the scene, a colour histogram thresh¬ 
olding method (as applied, for example, in 0 ) may be sufficient. Due to the 
variation in colour and texture of the plants and background in the image sets 
we are using, a pixel classifier using a Support Vector Machine trained on man¬ 
ually labelled images was applied for the mask generation. 


3.2 Calibration 

Camera calibration is achieved using a calibration object that displays known 
patterns to a variety of viewpoints. Rather than using independent planes as 
proposed in m and [22], a single rigid object is favourable here as it does not 
require that fixed cameras view planes in common. Such an object can also 
be placed within an automated greenhouse system so that calibration can be 
periodically performed or verified. The shape of the object is recorded in a file 
such as may be sent to one of the many acrylic laser cutting services so it can 
be rapidly constructed anywhere in the world. 

QR codes are used as calibration patterns as they are rich in features and 
can be uniquely identified. The patterns are printed onto durable adhesive labels 
for robustness against humidity and temperature. The adhesive labels are placed 
onto the object manually, resulting in some ambiguity in their true locations. 
Rather than rely on a large number of manual measurements, adhesive label 
placements are described by calibration object parameters which are estimated 
as part of the calibration process. 

Initial camera poses and intrinsic parameters are estimated assuming ideal 
(known) placements of calibration patterns. Subsequently, both camera param¬ 
eters and calibration object parameters are refined so as to minimise the sum of 
squared reprojection distances and error terms based on prior estimates of the 
calibration object parameters. Figure [3] shows the calibration object. 

3.3 Database Building 

The goal of the method is to generate a plausible plant model given the sil¬ 
houette in each view. To generate candidate models representing plausible plant 
structures, we use a database of pre-defined leaf models. These models are man¬ 
ually constructed using an interactive tool. Plants are modelled by specifying 
a series of 3D point locations tracing the axis of each leaf. To specify a point 
location, the user first selects a point on a leaf in one view of the plant, then 
selects the corresponding point in a second view. The selected point in the sec¬ 
ond view is constrained to lie on the corresponding epipolar line. The database 
currently contains models for 480 leaves, modelled from 230 plants. Each leaf is 
modelled with an average of 8 points. Figure [4] shows two views of a manually 
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Fig. 3. The calibration object Fig. 4. A plant model manually constructed from 

two views 



(a) Input (b) Classified (c) Skeleton 

Fig. 6. 2 views of the 3D tip and base 
Fig. 5. Extracting a skeleton for a frame P om t s 


modelled plant. To increase the density of the database, additional leaf models 
are generated by transforming modelled leaves to stretch their shape in multi¬ 
ple directions within a small distance range. This generates 100 models for each 
modelled leaf. 


3.4 Skeleton Extraction 

A set of 2D skeletons extracted from the silhouette for each view are used as 
estimates of the projection of the set of 3D leaf axes. An example of such skeletons 
being used for plant reconstruction is given in the reconstruction method of [23], 
where matching between skeleton points in orthographic images is used to recover 
3D leaf paths. To generate the skeletons, we use the thinning algorithm of [24] . 
An example of a skeleton extracted from a silhouette is shown in Figure [5] 


3.5 Leaf Tip Detection 

To limit the number of candidate models which need to be evaluated to find a 
model which corresponds to the current image set, information extracted from 
the 2D skeletons is used to guide the generation process. To identify possible 
tip points, we first construct a graph from each skeleton image. As some plant 
regions in the silhouettes may be disconnected due to sections of the plant which 
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are too thin to be detected, edges are added between nearby points to connect 
these isolated regions in the graph. 

From the graphs, we extract a set of possible 3D leaf tip points and a base 
point. We significantly reduce the space of possible models by considering only 
candidate leaf models with ends corresponding to these tip and base points. For 
each graph, we first detect a set of 2D points possibly corresponding to leaf tips 
by measuring the distance to the graph centre for each node and finding local 
maxima for this distance. These 2D points are matched between images to give 
possible 3D leaf tip locations. Matches for a point are found by locating points 
close to the corresponding epipolar line in a second view. 3D tip points are then 
determined by triangulation. Matches in further views are located by finding 
points close to the projection of the 3D points. The final position for each point 
is determined as the 3D point minimizing the sum of squared distances to the 
corresponding 2D points in all views. We select the 3D point closest to the input 
pot centre position as the base point. The selected base point and set of possible 
tip points for a plant are shown in Figure [6j 

The boundary of the extracted silhouette may not be smooth due to mis- 
classified background pixels. This results in extreme points in the graph which 
do not correspond to leaf tips, and 3D tip points being generated corresponding 
to points part way along the leaves. While such points could be removed using 
morphological filtering operations, doing so also eliminates important structural 
information. 

Instead, we use the set of graphs to detect 3D tip points which are likely to 
be part way along the path to a true leaf tip. We find the shortest path in the 
graph from each 3D tip point to the base point in each view where that point is 
visible, and remove any point for which these paths do not include at least 150 
pixels not included in the paths to a tip point farther from the base. Points with 
2D projections which are not close to the silhouettes are also rejected. 

3.6 Leaf Generation 

For each possible tip point, we build a set of candidate leaf models. Leaves from 
the database of manually modelled plants are linearly transformed to fit the tip 
and base points for each leaf to the selected tip point and base point positions. 
The transformed leaves are then evaluated against the images. The leaf models 
which best match the images are determined by measuring the distance in each 
image between sampled points on the models and the nearest point on the 2D 
skeleton for that image. 

The tip point, base point, and the orientation vector determined in the cali¬ 
bration process are used to define a linear transform mapping the base and tip 
of each leaf chosen from the database to the corresponding points in the current 
scene. This transform can then be used to map all points of the leaf into the 
scene. To efficiently evaluate distances from model points to skeleton points, a 
distance transform is applied to the 2D skeleton in each view, assigning each 
pixel in the image the distance to the nearest skeleton point. As leaves can hang 
over the edge of the pot, where they cannot be seen by some cameras, we also 
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(a) with penalty (b) without penalty 
Fig. 7. The effect of the curvature penalty 


make use of a 3D model giving the approximate structure of the pot and pot 
holder. This allows occlusion to be estimated and incorporated into the recon¬ 
struction. After evaluating the full set of transformed leaf models against the 
images, parameters for the best models are refined to improve their fit to the 
image set, as we do not expect the database to contain an exact match for each 
leaf. 

To refine the leaf while preserving its shape, we model each leaf using cubic 
B-splines bc{t) $i 3 , t G (0,1) parameterised by a set of control points C. The 
control points are optimised with respect to S = where S v = [sj,..., s^] 

is the set of skeleton points in view u, by minimising 

d (S> c ) = ^2[ dt+ f cc(t)dt (1) 

„ Jo y„<M&c W) Jo 

where the residual 


r„(x) = o„(x) (^min ||sj - A„x|| 2 J (2) 

measures the distance between the projection of a point on the leaf against the 
closest skeleton point in view v. Here, A v is the projection matrix for view v 
and o v (x) is a delta function that is 0 if x is occluded in view v and 1 otherwise. 
Residuals are inversely weighted by the number of views where a point is visible, 
to avoid biasing the optimisation towards a better fit for points which are visible 
in more views. To prevent significant changes in the leaf shape, the term 

c c (t) = a(K C {t) - Kc 0 it)) 2 (3) 


is added to the residuals to penalise changes in curvature with respect to the 
control points Cq of the original curve. The term 


KC (t) 


ll^cCOII 3 


(4) 


measures curvature of the B-spline be evaluated at t. The effect of the curve 
penalty on the reconstruction is illustrated in Figure [7} In both cases the opti¬ 
misation began from the curve illustrated in Figure ufa). Without the penalty, 
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Fig. 8. Refining parameters for a leaf model 



(a) Initial set (b) Optimised set 


Fig. 9. Initial and optimised set of leaf candidates 


the different parts of the hypothesised curve latch onto different, disjoint leaves 
in the image (Figure (7^b)). 

The curvature of a leaf may not be continuous, particularly where the leaf 
meets the stem. We therefore find any points in the 3D path of the original leaf 
model where a sharp change of angle (> 45°) occurs, and model the path as a set 
of one or more connected splines, with discontinuous curvature at these points. 
The number of control points for each segment is determined from the segment 
length. 

To fit a leaf to the image set, the position of the 3D control points are refined 
by applying Levenberg-Marquardt optimisation to a set of points sampled along 
the leaf. In practice, we define n points that are separated by approximately 
7.5mm along the original curve and define the distance residual in [2] by the 
distance transform over the skeletonised observation. The change in the shape 
of a leaf during refinement is illustrated in Figure [8] 

The distance measure is used to rank the full set of leaf models generated from 
the database. The best 200 leaf models are then selected and refined. Figure [9] 
shows a set of initial candidate models obtained for a point, and the same set of 
models after refinement. 

3.7 Structure Estimation 

The above process generates a set of possible leaf models which may be combined 
into a full plant model. In generating the complete plant model, multiple can¬ 
didates for each tip are tested, because overlapping leaves may result in several 
plausible paths from a tip to the base. For each tip point, we select 5 candidate 
leaves using the distance measure evaluated for the refined leaf. As multiple leaf 
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Fig. 10. The interior, exterior and overlapping areas of the set metric 

models may converge to the same shape in refinement, additional leaf models are 
not selected if there is only minimal deviation from an already selected model. 

On the basis of the leaf hypothesis set, and the anatomy-based prior which 
describes the ways in which such leaves may be combined, it is possible to con¬ 
struct a set of full-plant hypotheses. This process may be seen as a data-driven 
means of exploiting a generative model in a situation where sampling from a full 
generative model directly would be too computationally expensive. The genera¬ 
tive model for a plant such as wheat is relatively simple, but nonetheless far too 
complex to be sampled from directly. 

Each hypothesised structure is evaluated against the number of skeleton pix¬ 
els covered by the model, the number of pixels outside the plant which are 
covered, and the number of leaves used. Let X v be the set of skeleton pixels in 
view v. The set of ‘interior’ pixels which are supported by the set of leaves C is 
given by 


i v (C) = {i | (i G T v ) A (a v (i,£) > 0)} 


(5) 


where 



Le£ 


and 


{ 


f 1 if min t ||i - A v 6 L (t)|| < r 
0 otherwise. 



(7) 


counts the number of leaves which project to the pixel i G T v within a tolerance 


r = 10 pixels. This threshold helps to account for divergence between the skele¬ 
ton extracted for each frame and the projection of the true axis of each leaf. The 
set of ‘exterior’ pixels 


e v{£) = {j I (j e n) a (min||i—j|| > r)} 


(8) 


are in the image 7 Z generated by rendering C with projection matrix A v but are 
not within the threshold distance of any skeleton pixels. The quality of the leaf 
set is 


q(C) = ^2 (|4(£)| - P\ e v(£)\ - jo v (C)) 


vev 


(9) 
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where (3 controls the penalty for covering exterior pixels and 7 with 

Ov(£)= (««(i>£)-l) (10) 

i ei v (£) 

penalises solutions where multiple leaves overlap the same set of pixels. Conse¬ 
quently, ([9]) favours models that closely match the skeletons in each view while 
using the smallest number of leaves. Figure [lO] illustrates the segmentation of the 
observed image into interior, exterior and overlapping pixels given a hypothesised 
leaf set C. 

A set of leaves C is chosen from a larger set of candidate leaves by a random, 
greedy search. Let C l = {C[, ..., C^} be the set of n candidate control point 
sets for leaf tip l. A leaf model C' is randomly chosen and removed from the set 
V = {C l }\/i. If q(C U {C'}) > q(C), then C' is added to the initially empty set of 
hypothesised leaves C and C l —>> 0 where C l is the set of candidate leaves that 
contained C'. The process of sampling leaves from V and adding them to the 
model set continues until V is empty. 

4 Results 

This method has been tested on a set of plants with up to 8 leaves each, with 
manual measurements taken for the first 4 leaves of each plant. Figure [12] shows 
the original images from two cameras, and the reconstructed plant model pro¬ 
jected into those images, for 6 plants. These results show the structure of the 
plant being accurately recovered despite overlap between multiple leaves. These 
results were generated with 50000 runs of the model generation process, and 
with weights set to a = 2 x 10 -7 , /3 = 1.4 and 7 = 0.3. Result images for more 
plants are included in Online Resource 1. 

In most cases, the reconstruction process determined the correct number of 
leaves and generated a model close to the true shape of the plant. Figure [Tl] 
shows some cases where a leaf was not reconstructed, or was only partially 
reconstructed. In Figure [llja), a leaf was not reconstructed due to the leaf tip 
and most of the length of the leaf being occluded in all views by the leaf labelled 
in red. In Figure [TTJb) , only part of the shape of the leaf labelled in yellow 
was recovered, as a close match for the leaf was not found in the database. 
This limitation would be improved with a more comprehensive model database. 
A leaf model was not fitted to the full extent of the leaf labelled in green in 
Figure [lljc), due to the pixels of a dead leaf tip being classified as background 
during silhouette generation. 

For this set of plants, we have compared leaf length measurements automat¬ 
ically extracted from the models with manual measurements of the first 4 leaves 
of each plant. Manual measurements were taken from the leaf tip to the point at 
which the leaf meets the stem. To measure this distance from the reconstructed 
leaf models, we estimate this point by finding the point at which overlapping 
leaf models diverge. Table [l] shows automatically and manually measured leaf 
lengths in millimetres and relative percentage error for the set of plants seen 
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Fig. 11. Partially reconstructed leaves 


in Figure [12] For tests on a set of 40 plants, the average difference between the 
manual measurements and our estimated leaf lengths was 19.06mm. The average 
relative error was 8.64%. 

This testing has highlighted an unforeseen ambiguity in the (stem-side) end 
point of such leaf measurements which leads to differences between the manually 
measured quantity and that estimated from the recovered structure. It also in¬ 
dicates a need to conduct repeated manual measurements so as to estimate the 
error in that process. Despite these limitations, the results show that the method 
is capable of automatically recovering meaningful plant structure estimates from 
image sets. 

Figure [13] shows results of applying this method to more mature plants with 
a greater density of leaves. In these cases, the structure of the majority of leaves 
was still recovered. However, some leaves with tips in regions where structure is 
dense were not identified, and the accuracy of the curves for the reconstructed 
leaves was also lower in these regions. Improving reconstruction accuracy for 
more mature plants will be a focus of further development of this method. 


Table 1. Measurement results for the first 4 leaves 


Manual (mm) 
Estimated (mm) 
Relative (%) 

150.64 

147.0 

2.42 

Plant 1 

220.68 299.53 

216.79 299.89 

1.77 1.55 

245.26 

241.99 

0.92 

138.74 

145.99 

5.23 

Plant 2 

243.89 332 

214.75 292.73 

11.95 11.83 

351 

337.99 

3.71 

Manual (mm) 

144.97 

Plant 3 

263.75 378 

224.13 

115.73 

Plant 4 

203.23 279.82 

320.0 

Estimated (mm) 

145.91 

259.87 

376.73 

242.94 

137.0 

200.99 

279.0 

287.92 

Relative (%) 

0.65 

1.47 

0.34 

8.39 

17.51 

1.1 

0.29 

10.02 

Manual (mm) 

101.4 

Plant 5 

185.82 259.16 

299.87 

119.22 

Plant 6 

211.86 273.85 

304.55 

Estimated (mm) 

117.0 

162.0 

255.81 

251.98 

130.99 

184.51 

272.54 

265.98 

Relative (%) 

15.38 

12.82 

1.29 

15.97 

9.87 

12.91 

0.48 

12.66 
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Fig. 12. Original images and reconstruction results 
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Fig. 13. Reconstruction results for more mature plants 


5 Conclusions and Future Work 

We have presented a method suitable for recovering the structure of thin plants 
from a small set of images captured by widely spaced cameras. There are a range 
of potential future developments for this method. Although the present method 
operates only on RGB images, it would be straightforward to incorporate depth 
map information into the fitting process, allowing for reconstruction using depth 
camera or laser data from a limited range of views. 

The method could potentially be applied to single images, using the variation 
in plausible reconstructions of the image to determine the range of possible 
values for various plant properties. The method could also provide a means 
of estimating further physical properties of leaves from measured properties of 
leaves represented in the database. The structure estimates can be used for leaf 
angle and length measurements, and we plan to use these paths as a basis for 
also measuring leaf width and senescence. 

We plan to use the estimated structures of plants over time to track plant 
growth, with a database of models of developing plants used to determine plau¬ 
sible matches between the estimated leaves at different time steps. The method 
will also be refined to improve the reconstruction accuracy for more mature 
plants, where the structure of individual leaves is more difficult to distinguish 
using only skeletons extracted from silhouettes. 
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